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ABSTRACT 


Detecting eye blink and yawning is important, for example in systems that 
monitor the vigilance of the human operator, eg; Driver's drowsiness. Driver 
fatigue is one of the leading causes of the world's deadliest road accidents. 
This shows that in the transport sector in particular, where a driver of heavy 
vehicles is often open to hours of monotonous driving which causes fatigue 
without frequent rest periods. It is therefore essential to design a road 
accident prevention system that can detect the driver's drowsiness, determine 
the driver's level of carelessness and warn when an imminent danger occurs. 
In this article, we propose a real-time system that uses eye detection 
techniques, blinking and yawning. The system is designed as a non-intrusive 
real-time monitoring system. The priority is to improve driver safety without 
being intrusive. In this work, the blink of an eye and the driver's yawn are 
detected. If the driver's eyes remain closed for more than a certain time and 
the driver's mouth is open to yawning, the driver is said to be fatigue. 
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I. INTRODUCTION 

Detecting eye blinks is important for instance in systems that monitor a human 
operator vigilance, e.g. driver drowsiness [1], in systems that warn a computer 
user staring at the screen without blinking for a long time to prevent the dry eye 
and the computer vision syndromes [2], 
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In human computer interfaces that ease communication for 
disabled people, or for anti-spoofing protection in face 
recognition systems [2]. In this paper, we are focusing on 
designing of a system that will monitor the open or close 
state of driver's eyes and yawning in real time. 

The techniques of drowsiness/fatigue detection can be 
broadly classified into three major categories: Physiological 
measures, indirect vehicle behavior and directly observable 
visual behaviors [6], [7]. The best detection accurate 
techniques are based on physiological phenomena like brain 
waves, heart rate, pulse rate and respiration. These 
techniques are intrusive, since they need to attach some 
electrodes to the drivers, causing annoyance to them. 

Driver drowsiness is one of the biggest safety issues facing 
the road transport industry today and the most dangerous 
aspect of driver fatigue is falling asleep at the wheel [4]. 
Fatigue leads to sleep, it reduces reaction time (a critical 
element of safe driving]. It also reduces vigilance, alertness 
and concentration so that the ability to perform attention- 
based activities (such as driving] is impaired. The speed at 
which information is processed is also reduced by 
sleepiness. The quality of decision-making may also be 
affected [5]. 

Developing vision based warning systems for drivers is an 
increasing area of interest. Computer vision has gained a lot 
of importance in the area of face detection, face tracking, eye 
detection, Yawning detection for various applications like 
security, fatigue detection, biometrics. 

Driver falls in micro sleep, results in collision with object or 
vehicle, or they cannot recognize that he or she has drifted 
into a wrong lane. The consequences of a drowsy driver are 


very dangerous and lead to loss of lives, casualties and 
vehicle damage. 

The purpose of this paper is to develop a system to detect 
eye blinking and yawns. Emphasis will be placed on 
designing a system to accurately monitor the open or closed 
state of the eyes and the yawning conditions in real time. By 
monitoring the eyes and mouth, it is believed that the 
driver's fatigue symptoms can be detected early enough to 
prevent a car accident. Fatigue detection involves a sequence 
of images of the eyes and mouth, as well as the observation 
of eye movements and movements. It also implies the 
yawning state of the mouth. 

This paper focuses on the position of the eyes, which consists 
of searching for the entire eye image and determining the 
position of the eyes using a self-developed image processing 
algorithm. Once the eye position is identified, the system is 
designed to determine if the eyes are open or closed and to 
detect fatigue of driver. 

II. METHODOLOGY 

In this paper, we represent a methodology for detection of 
eye blinking and yawning robustly in real time environment. 
The condition of closed eyes and open mouth to observe 
yawning are detected simultaneously. This is continuously 
capturing the real time video using webcam and Viola Jones 
Algorithm is used to detect the eye and mouth. This work is 
based on the combination of yawning and eye blink 
detection technique. This work is composed of various steps 
as under: 

> Face, eyes and mouth detection 

> Eye blinking detection (open/close] 

> Yawning detection (open] 
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A. Video Frames 

When the system is initiated a delay of two to three seconds 
is experienced for capturing the image for the first time. This 
results in losing the data from the first three frames. The 
segment of video thus obtained from the camera is then used 
to extract each segment of video frame. The system runs at 
25-30 frames per second for detecting driver's drowsiness 
considering eye and yawning in the real time. This 
application is developed in MATLAB in Windows 
environment with webcam. 

B. Face, Eye and Mouth Detection 

The system actually starts its execution once the face has 
been detected. The images obtained from the frames are now 
subjected to face detection part of the system. The most 
commonly used algorithm for face detection is the Viola 
Jones algorithm. This method focuses on detecting objects in 
images. Object detection is done by Simple rectangular 
features, called Haar-like features, an integral Image for 
rapid feature detection, AdaBoost machine-learning method 
and cascaded classifier to combine many features efficiently. 
To detection of face and eyes using 'Viola Jones' method with 
a training set of faces and eyes provided is taken. Since we 
already get face center, we can estimate the position of eyes 
based on the common fact that human's eyes are located in 
the top half of the face. Considering the center of the eye, and 
taking ROI of the eye with the assumption that both eyes 
always blink simultaneously, which can detect the motion of 
single eye. According to the estimated position, a small 
rectangle around the center of eye is drawn, and creates a 
corresponding cv::Mat matrix as the eye ROI, for the use in 
step2. 

C. Viola -Jones's Object Detector 

Face detection is done by Viola-Jones method developed by 
Paul Viola and Michael Jones. This method focuses on 
detecting objects in images. Object detection is done by 
Simple rectangular features, called Haar-like features, an 
integral Image for rapid feature detection , AdaBoost 
machine-learning method and cascaded classifier to combine 
many features efficiently[8]. These methods are described as 
follows. 

1. Haar-Like Features 

Haar-like feature considers adjacent rectangular regions at a 
specific location in a detection window. It sums up the pixel 
intensities in each region and calculates the difference 
between these sums. This difference is then used to 
categorize subsections of an image. The key advantage of a 
Haar-like feature over most other features is its calculation 
speed. Due to the use of integral images, a Haar-like feature 
of any size can be calculated in constant time (approximately 
60 microprocessor instructions for a 2-rectangle feature). 


(c) Foiu-rectangle feature? 


Figure 2 Haar-like features 

2. Integral Image 

The simple rectangular features of an image are calculated 
using an intermediate representation of an image, called the 
integral image. The integral images are an array which 
consists of sums of the pixels' intensity values located 
directly to the left of a pixel and directly above the pixel at 
location (x,y) inclusive. Here, A[x,y] is the original image and 
Ai[x,y] is the integral image [8]. 

Ai[x,y] = ^A[x,y],x < x, y'< y 

3. AdaBoost 

Adaboost, nothing but "Adaptive Boosting ", is a machine 
learning method given by Yoav Freund and Schapire in 2003. 
It can be used with many other types of learning algorithms 
to improve their performance. Adaboost takes a number of 
positive and negative images features and training sets, The 
machine creates a set of weak classifiers of Haar-like 
features. It selects a set of weak classifiers to combine and 
that assigns lesser weights to good features whereas larger 
weights to poor features. This weighted combination gives 
strong classifier. 

4. Cascaded Classifier 

The cascade classifier consists of number of stages, where 
each stage is a collection of weak learners. The weak 
learners are simple classifiers known as decision stumps. 
Boosting is used to train the classifiers. It provides the ability 
to train a highly accurate classifier by taking a weighted 
average of the decisions made by the weak learners. 




Figure 3 Cascades of Classifiers 


Each stage of the classifier shows the region defined by the 
current location of the sliding window as either positive or 
negative. Positive indicates an object was found and negative 
indicates no object. If the label is negative, the classification 
of this region is complete, and the detector shifts the window 
to the next location. If the label is positive, the classifier 


@ IJTSRD | Unique Paper ID - IJTSRD28004 | Volume - 3 | Issue - 5 | July - August 2019 


Page 2461 
















International Journal of Trend in Scientific Research and Development (IJTSRD) @ www.ijtsrd.com elSSN: 2456-6470 


passes the region to the next stage. The detector reports an 
object found at the current window location when the final 
stage classifies the region as positive. It is used to eliminate 
less likely regions quickly so that no more processing is 
needed. 

D. Eye blinking (open/close) detection 

The template matching method is used to detect the eye is 
open or close. The various conditions of eye images in 
database are correlated with detected eye image. The 
template-based methods search for eye image matched to a 
template of generic eye model designed by eye shape. Since 
people always blink both eyes at the same time, so in this 
phase, only right eye is monitored. To detect eye blinking, 
the current state of the eye is needed as either open or 
closed. If the state of eye changes from closed to open, it 
indicates an eye blinking. If the state of the eye keeps closed 
for a certain amount of time, the eye will be detected as 
closed. 


A ring buffer with a length of 100 is considered in this work 
here each frame, if eye blinking is detected, 1 otherwise 0 is 
written into buffer. So, when the system warms up (after the 
very first 100 frames), it will calculate the eye blinking rate, 
and keeping update at every frame. 


E. Yawning Detection 

The yawning motion is detected by correlating the images of 
mouth in database. The various stage of captured mouth 
models are collected in database. And then template 
matching method is used to detect the open or close 
condition of mouth. With matching method, it is very easy to 
decide if a person is yawning or not by checking the stage of 
the mouth keeps open for a certain amount of time, and this 
is a way we do it. 


III. Experimental Results 

The performance of eye blinking and yawning detection 
system has been measured under different conditions. In this 
work, eye blinking rate and yawning is measured to detect 
drowsiness. Because when someone felt sleepy at that time 
his/her eye blinking are different from normal situations and 
yawning often so that easily detect drowsiness. Fig. 4 to 7 
shows the eye blinking and yawning based drowsiness 
detection. First, the face is detected and tracked in the series 
of frame shots taken by the camera. In this system the 
position of eyes and eye states are monitored through time 
to estimate eye blinking frequency. The next step is to detect 
and track the location of the mouth. After detection of the 
mouth, the yawning state is detected. 



Figure 3 Screenshot showing the eyes open and mouth 
closed condition 



Figure 4 Screenshot showing the eyes open and mouth 
open condition 






Figure 5 Screenshot showing the eyes close and mouth 
open condition 
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Figure 6 when both eyes are closed and mouth are open 
frequently showing the warning message " Fatigue " 

IV. Conclusion 

In this article, we presented a low cost yet effective real-time 
blink and yawning detection system. Through our extensive 
experimental results, we find that using the Haar classifier 
and the template-matching method has satisfactory 
detection speed and accuracy. This application can be 
implemented in the real time to reduce traffic accidents rate 
due to drowsy drivers and it can also help drivers to stay 
awake when driving by giving a warning when the driver is 
sleepy. 
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